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Abstract 

This paper discusses the advantages and disadvantages of administering a survey 
questionnaire via the Internet (i.e., utilizing both Web-based and e-mail platforms). 
Highlighted are problems associated with the use of listservs, e-mail addresses, formats 
for downloading and compiling data, and the nonrandom nature of the resulting 
sample. 
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Lessons Learned from the Administration 
of a Web-Based Survey 



Introduction 

Web surveys are an extremely promising method of data collection (Schillewaert, 
Langerak, & Duhamel, 1998). Witte, Amoroso, and Howard (2000) indicated that, 
Internet research is "an area marked by great potential but also little experience." 
Advantages of Web surveys include a high rate of response, short time frame for the 
collection of responses, and time and cost savings. The World Wide Web certainly 
addresses the need for a less expensive and more expedient method of data collection 
(Schillewaert, Langerak, & Duhamel, 1998). Furthermore, Schillewaert, Langerak, & 
Duhamel (1998) have identified several additional benefits of using the Web for data 
collection. These benefits include a faster response, easy transfer of data into a database 
for analysis, cost savings, convenience for the respondent, and the possibility of wider 
geographic coverage. 

As with any method of data collection, there also exist disadvantages. These 
include the potentially nonrandom nature of the sample, unavailability of population 
lists, computer access to the survey, and various technology-related issues. Often, the 
results of Web-based surveys differ when compared to written questionnaires and 
telephone surveys (Taylor, 2000), although Saphore (1999) found that there were no 
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differences in the pattern of responses between a Web survey and an identical pencil- 
paper form of the same survey. Furthermore, he concluded that there were no 
differences in the psychometric qualities of the two forms of the survey. Similar to the 
relative benefits, Schillewaert, Langerak, & Duhamel (1998) have identified several 
limitations of using the Web for data collection. These limitations include the inability to 
clearly define the population, lack of technological familiarity on the part of 
respondents, the potential for being able to identify respondents, and browser 
incompatibility problems. 

One of the most substantial concerns about Web surveys is the nonrandom 
nature of the respondent group (Witte, Amoroso, and Howard, 2000). However, the 
issue of nonrandomness is not unique to Web-based survey research and will likely 
decline as the use of this methodology permeates further into the society of researchers 
(Witte, Amoroso, and Howard, 2000). Taylor (2000) has suggested that we remember 
that online data collection is not based on probability sampling, but rather on 
"volunteer" or "convenience" sampling. Nonetheless, Young and Ross (2000) state that 
the use of the Internet to collect data may be one of the most profound developments in 
survey research. 

Even though their use is on the rise, not much is known about the relative 
effectiveness of Web-based surveys, especially in the field of social science research. 
More research is needed regarding their efficiency, as well as their limitations. This 
paper attempts to document some of the advantages and limitations of data collection 
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via the use of Web-based surveys, specifically from the perspective of one such survey 
research study which utilized this methodology. 

Purpose 

This paper chronicles the methodology utilized in a research study involving the 
collection of data via a Web-based survey, focusing on the advantages and limitations of 
the methodology. 

Method 

The primary information for this paper was collected through the administration 
of a Web-based survey. The study involved the administration of the Teacher Motivation 
and Job Satisfaction Survey to K-12 teachers (see Figure 1; the survey can also be viewed 
online at http://versonnl.bgsu.edu/~mertler/TMJS-Survey/TMlS.html ). 



Insert Figure 1 about here 



Many of the difficulties encountered by the researcher occurred during the 
planning phase of the study, as opposed to the actual data collection phase. Therefore, it 
is important to outline the specific steps taken in conducting this study. Initially, 
teachers had to be notified of and directed to the Web site containing the survey. Since 
individual email addresses were not available and the desire was to obtain a large 
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sample, it was decided to attempt posting messages to listservs, or electronic discussion 
groups, which would direct individual teachers to the survey. Over several days, 
searches for educational listservs were conducted on the Web. Initially, 43 listservs were 
identified. Only 13 of these permitted the posting of the cover message (see Figure 2) to 
the entire listserv directing teachers to the survey. The 13 listservs who participated in 
the study are listed in Figure 3. 



Insert Figure 2 about here 



Insert Figure 3 about here 



The survey was "housed" on a Web site called "Digisurveys.com" 
( http://www.di 9 isurveys.com ). Upon completing the survey, teachers simply clicked the 
"Submit" button at the bottom of the page. They were transported to a "Thank you" 
page and their responses were submitted electronically in CGI script to a predetermined 
email address, where they were received as a text file. Each individual file was then 
filtered through a software program called Eform into a data stream and formatted into 
columns, which were then converted into ASCII format. The webmaster of 
Digisurveys.com the provided the researcher with the ASCII file on disk, who then 
converted it directly in an SPSS data file. 
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Limitations of Web-Based Surveys 

Several limitations of this specific methodology arose during the process of 
conducting this particular study. Four primary limitations are listed and discussed 
below. 

> Listservs and Email Addresses 

The first major limitation involved the use of listservs, which was necessitated due 
to the lack of individual teacher email addresses. First, all listservs are subscription- 
based— that is, one cannot post a message to a listserv without first becoming a 
member of the list. This is typically not a problem; one simply sends an email 
message to the administrator of the listserv requesting membership. Within a day 
or two, confirmation of membership was received from all 43 listservs. 

In an effort to save time, the first message posted by the researcher was sent to all 
43 listservs simultaneously using a university email account (in other words, all 43 
email addresses were entered in the "To:" line of the email message, separated by 
commas). This created serious problems. Within minutes, return messages were 
received from several of the listservs, each containing a message similar to the 
following: 

"We have detected multiple simultaneous email messages sent from this address. We 
assume that you are trying to "spam" our listserv membership. You have been 
banned from posting any messages to this listserv for a period oftivo months. " 

Obviously, this created some serious problems for the study at hand. Due to this 
banning, the initial cover message would not be forwarded to the individual 
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listservs. The researcher was forced to create a Web-based email account for the sole 
purpose of conducting the study. The cover message was then successfully sent out 
to all listservs individually. 

Everything seemed to be proceeding alright until a new wave of return email 
messages began arriving. The researcher then discovered that the administrators of 
several of the listservs had to review the content of the message prior to it being 
posted to everyone. After several days, messages were received from a subset of 
those stating that this was the type of request that they would not post to their 
respective listservs due to the fact that the memberships did not appreciate being 
inundated with this type of request. A beneficial aspect of joining the listservs is 
that one knows when your message has been posted since you also receive it as a 
member of the list. Some of the listservs neither posted the message nor contacted 
the researcher; they just simply chose not to post it. After all of these unanticipated 
roadblocks, 13 of the initial 43 listservs posted the message to their entire 
membership list. 

> Umitatmts of Technology 

When this study was originally planned, it was the intent of the researcher to 
create a university account in order to electronically receive all of the teachers' 
responses. Upon making that formal request, it was discovered that Information 
Technology Services at the institution no longer provided CGI accounts due to the 
fact that students and others had learned how to "hack" into those accounts. 
Therefore, the researcher had to look elsewhere. There are several sites on the 
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Web — such as "zoomerang.com" and "survey.com" — that allow you to customize 
your own survey and house it on their servers. Fortunately for the researcher, one 
such site — "Digisurveys.com" — is maintained by a former graduate student. All 
services — uploading the survey, writing the code for the relay of responses, and 
conversion into an ASCII file — were provided for a nominal fee. 

This type of service can make the retrieval of data quite easy. However, one must 
make sure, at the outset, to carefully specify the format in which the data is to be 
received. 

> Lack of_a Population List 

Since the researcher desired to obtain a very large sample, and since individual 
email addresses were not available, an alternative had to be found. The idea of 
using listservs was a good one, from both logistical and practical viewpoints. 
However, one can not be sure of the population that is being reached with the 
request to complete the survey posted on a listserv. For example, the population of 
interest for the study at hand was K-12 teachers. Some teachers were automatically 
excluded from the opportunity to complete the survey due to the fact that they were 
not members of the identified listserv, that the listserv administrator refused to 
allow the cover message to be posted, that they did not have an email address 
(which is required for membership on a listserv), or that they did not have access to 
or comfort with a computer, among others. This fact has obvious consequences in 
relation to the representativeness of the resultant sample. 
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> Revresentativeness ofjhe Sample Data 

Since no population list was available, it follows that it was not possible to select 
a random, representative sample. At best, this methodology — as used in this 
study — resulted in merely a sample of convenience. Obviously, one needs to be 
cautious when using Web surveys, especially when respondents are obtained via 
listservs, about generalizing the results to a larger target population. 

For purposes of this study, the researcher was confident in many of the results 
since they mirrored results of a smaller pilot study. The smaller study utilized the 
same survey items as well as a more traditional pencil-paper format. 

Advantages of Web-Based Surveys 

Advantages of this methodology, which will be discussed thoroughly in the 
paper, include: 

> Number of Responses Received 

Although it was impossible to calculate a rate of return for the survey (without 
knowing the actual number in the population), the number of responses received 
exceeded the expectations of the researcher. In the fourteen days that the survey 
was "active" on the Web site, a total of 969 responses were received. Even if 
individual addresses had been available, one must question whether that size of 
return could have been achieved using more traditional survey methods. 
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> E fficiency o f Data Collection 

Difficulties typically associated with mail surveys encountered at both ends of 
the process are never realized when using this methodology. One does not have to 
factor into the process the amount of time it will take for the surveys to reach the 
intended respondent, nor the amount of time it will take for the survey to arrive 
back to the researcher. Additionally, there are no paper copies to keep track of or 
j store following receipt of the completed surveys. Finally, the endless hours spent 
stuffing envelopes is not an issue with which one must deal when using Web-based 
surveys. 

> NonzJssue of_Data Entry 

Probably the most meaningful time-saving feature of the Web survey is the fact 
that individual responses do not have to be manually entered into a data file. First, 

this reduces the chance of human error with respect to data entry. Second, manual 

\ 

entry of data is often a very time-consuming portion of the survey research process. 
However, the use of Web-based surveys all but eliminates this step in the process. 
In essence, each respondent "enters" his or her own responses into the data file 
simply by clicking on "Submit." Finally, even if computer scannable response 
forms are used, the Web survey process is still more efficient. 

> Monetary, Savings 

Finally, the use of Web-based surveys can result in substantial monetary savings 
for the researcher. If this particular study had been conducted using conventional 
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surveys methods, and if it had resulted in nearly 1,000 responses, the cost would 
have been substantial. Assuming that the cost for each survey would have been 
about $1 per person (this includes copying, postage, and envelopes) and a 
conservative return rate of 40%, the total cost of preparing, sending, and receiving 
the surveys would have been approximately $4,000. In contrast, this particular 
study cost the researcher $120. Additional savings could be realized if persons were 
employed for purposes of data entry and/ or statistical analyses. 

Discussion 

Even though their use is on the rise, not that much is known about the relative 
effectiveness of Web-based surveys. More research and related information— such as 
that provided in this paper — is desperately needed to assist researchers in determining 
the efficiency of Web surveys, as well as their limitations. There are certainly research 
design issues of a theoretical nature which must be examined, such as the issue of 
representative sampling and the resulting generalizability of the findings. These are 
issues that the research community must resolve because the use of Web-based surveys 
is increasing in popularity and use. In addition, there also exists a variety of practically- 
oriented issues with which researchers should be familiar prior to engaging in their own 
Web-based surveys. The purpose of this paper was to share with survey researchers 
many of the unanticipated barriers encountered in this particular study, as well as many 
of the distinct advantages associated with the use of this methodology. Hopefully, this 
paper has shed some light on several of those important issues. 
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Figure 1. 

The Teacher Motivation and Job Satisfaction Web-Based Survey 
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Figure 2. Cover Message for The Teacher Motivation and Job Satisfaction Survey 



Hello, Everyone! I am currently conducting an online survey 
research study titled "Teacher Motivation and Job Satisfaction 
of Public and Private School Teachers", the purpose of which 
is examine the current levels of teacher motivation and job 
satisfaction for teachers. 



1 



The purpose of this email message is to ask for your 
participation in the study. The survey will only take about 
4-6 minutes to complete. When you have completed the survey, 
simply click on the SUBMIT button to send your responses, to 
me. Please make sure you submit your responses only once! 

Please be assured that your responses will be anonymous and 
confidential. There will be no way for me to determine the 
origin of your responses. You will not be contacted for any 
further information. Additionally, no individual information 
will be shared; only aggregate results will be reported. 
Finally, due to the web-based nature of the survey, there 
exists a minimal chance that your responses could be 
intercepted while being transmitted. 

Your participation in this study is voluntary. By completing 
and submitting the survey, you are giving your consent to 
participate. If you do not wish to participate, simply 
disregard this message. If you have any questions regarding 
this survey study, I may be contacted at 
mertler0bgnet . bgsu . edu . 

I would like very much for you to participate in the study by 
completing the brief survey which can be found at: 

digisurveys . com/Mertler 

Please pass this message on to other teachers you know or 
direct them to the website. In advance, thank you very much 
for your participation in this research endeavor and best of 
luck in the remainder of your school year! 



Best Regards, 

Craig A. Mertler, Ph.D. 
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Figure 3. List of Participating Listservs 



1. Special Education Discussion List S^ECEDzLRLISTSERy_. UGA^EDU 

2. AERA-K Division K: Teaching and Teacher Education AERAz 
Kgasu . edu 

3. AERA-C Division C: Learning and Instruction AERAzC@a_su_. edu 

4. Kentucky K-12 Arts and Humanities Teachers Discussion List 
KYARTS0LSV . UKY_. EDU 

5. KY Business Education Teachers KYBUSED^LSy_. UKY_. EDU 

6. Kentucky Geography Teachers KYGEOGjLSV UKY . EDU 

7. Elementary Education elemed£acpub_, duke_. edu 

8. K-12 Educators Interested in Educational Administration 
Kl2AmM^LISTSEPAL SYR^EDU 

9. Foreign Language Teaching Forum 
F2jTEACH^LI_STSERV_. ACSU_. BUFFALO • EDU 

10. Math Educators ma tjjsedzJJ^d^kJji . edu . a u 

11. English Teachers' List Secondary School level TEACHz£N£z 
MNETPALS_ l IjSOFT ■ COM 

12. For parents, teachers, and others concerned about 
education ABLETECH-LftLIST.SE.RV'. OKSTATE_. EDU 

13. Beginning Teachers Network List BTN@J2E£L QLQ_. GOy_. AU 
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